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Defining  the  solution  to  a stochastic  differential  equation  to  be 


the  solution  to  the  Martingale  problem  of  Strook  and  Varadhan,  we  obtain 


results  on  the  existence  of  an  optimal  stationary  control  for  the  average 


a necessary  and  sufficient  condition  for 


optimality  of  a control,  and  a number  of  other  related  results 
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1. 


Introduction 


The  purpose  of  the  paper  is  the  development  of  a necessary 
and  sufficient  "Dynamic  Programming"  like  condition,  for  the  average 
cost  per  unit  time  problem.  The  condition  is  similar  to  those 
developed  for  other  problems  by  Davis  and  Varaiya  [1]  and 
Bismut  [ 2 ] . In  addition  to  its  intrinsic  interest,  the  criterion 
appears  to  be  useful  for  the  problem  of  approximation  and  computa- 
tion (see  the  corollary  and  remark  in  Section  6).  The  main  Theorems 
are  3.1  (characterizing  the  invariant  measure),  4.4  (existence  of  an 
optimal  stationary  control),  5.1  (characterizing  the  auxiliary  VU  ( • ) 
function),  and  6.1  (necessary  and  sufficient  condition  for  optimality). 
Also,  a number  of  auxiliary  results  are  obtained. 

We  will  use  conditions  (Al)  - (A5) . 

(Al)  bet  a(')  denote  a bounded  uniformly  continuous  and 
uniformly  positive  definite  r x r matrix  valued  function  on  the 

If 

Euclidean  space  R . 

Let  ^ denote  a compact  convex  set  in  some  Euclidean  space  and 
which  contains  the  origin. 

(A2)  f(’),b(*,*),k  (*,*)  are  measurable  Rr,Rr,  and  R^  valued 

functions  on  Rr , Rr  x ^ and  Rr  x % , resp . ; b and  k are 
bounded,  and  are  continuous  in  their  second  argument  for  each  value 
of  the  first  argument,  and  b(x,0)  = 0.  f(-)  is  bounded  on  bounded  set' 

(A3)  The  set  { b (x , a ) , k (x  , a)  , a e } = (b  (x,  -2  ) ,k  (x,  %)  ) 


, If 

is  convex  and  compact  for  each  x e R . 


Any  measurable  ^ 

valued  function 

u ( • ) 

r 

on  R 

is  called 

an  admissible  control. 

Functions  b ( • , • ) 

and 

k ( • , • ) 

are  said 

to  be  admissible  if  they  satisfy  (A2) , (A3)  and  have  the  form 

b (x , u (x) ) , k (x , u (x) ) for  admissible  u(»).  We  will  often  write 
bU ( • ) = b(*,u(*)),  ku  ( • ) = k(-,u(-)).  Our  systems  model  is  the 
stochastic  differential  equation 
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(1.1)  dx(t)  = [ f (x ( t ) ) + bU(x(t))]dt  + a(x(t) )dw(t) , x(0)  = x, 


where  w(-)  is  a standard  Wiener  process.  In  particular,  the 
process  x(*)  will  be  defined  to  be  the  solution  to  the 
martingale  problem  of  Strook  and  Varadhan  [ 3 J;  hence  w(*)  may 
be  defined  implicitly  in  terms  of  x(-).  As  pointed  out  by 
Bismut  [ 2 ] , there  are  a number  of  advantages  to  using  the 
"martingale  problem  solution"  definition  of  (1.1),  particularly  when 
questions  of  existence  are  of  interest.  Here,  only  feedback 
(Markov)  controls  are  considered. 

The  cost  functional  is 


(1.2) 


0 (u) 


lim 
T ► ■ 


rT 

k (x  (s)  ,u  (x  (s))  )ds 

J 0 


k (x,u (x) ) u (dx) , 

J u 


where  u (•)  is  the  unique  invariant  measure  for  (1.1),  which  will 
exist  under  conditions  to  be  imposed.  Also,  denotes  expectation 

under  control  u(-),  and  initial  condition  x. 

In  order  for  the  problem  to  be  well  defined,  we  need  some  sort 

of  recurrence  for  each  control.  In  a sense,  we  will  assume  ((A4), (A5) ) 

that  the  effects  of  f(-)  dominate  those  of  b(*,*)  for  large  ix! 

and  all  u(*).  Assumption  (A4)  will  be  convenient,  and  (A5) , while 

avoidable,  does  provide  a relatively  simple  method  for  obtaining  some 

required  estimates.  Both  are  satisfied  by  a large  number  of  problems. 
(A4 ) There  is  a non-negative  twice  continuously  differentiable 

real  valued  function  ( • ) such  that  (x)  -*  °°  a_s  |x| 

and  for  some  e > 0 and  compact  K^, 

y UW^ (x ) < -c,  x / , all  admissible  u(*), 


(1.3) 


-3 


where 


^ ‘ sr^r:  + ? ‘V*’  + b>>>  drr  ■ 


if  D 


1 3 


the  differential  generator  of  (1.1),  and  a(*)  = a (•  ) a ' ( • ) /2  . 


(A5)  Let  W2(x)  = There  are  constants  > 0,  a > 0, 

such  that  for  all  admissible  u ( • ) , 


(1.4)  < ^2  - q2  (x)  , where  q2  (x)  > 0, 

and  q2(x)/W^(x)  > a > 0.  Let  ^ denote  a compact  set  such  that 
q2 (x)  > c2  for  x / K2 . 

Remark.  Suppose  that  f (x)  = Ax  and  ft  = f(x)  is 
asymptotically  stable.  Then  we  may  use  x'Px  = (x) , where  P 

satisfies  the  Liapunov  equation  A'P  + PA  = -Q  < 0.  Also,  (A5)  holds. 

For  some  additional  motivation,  let  us  consider  a Dynamic 
Programming  approach.  Suppose  that  there  is  a smooth  function 
V ( • ) and  a constant  y such  that 

inf  [Jz^UV(x)  + k(x,u(x))  - y]  = 0 
u (x)  c 

= A (x)  + inf  [V'  (x) b (x,u  (x) ) + k(x,u(x))  - y] , each  x. 
u(x)e:‘2f  X 


(1.5) 


If  the  solution  to  (1.1)  is  well  defined  for  u(x)  = u(x),  the 
minimizer  in  (1.5),  and  if 


E^V(x(t))/t  -*-0  as  t -*  °°, 


then 


lim  ^ E 

T->0O 


u 

X 


(x (s) ) ds 


J 0 


y. 


If,  in  addition,  E^V(x(t))/t  -*■  0 as 
Pu(x,t,-)  - u (•)  (PU(x,t,D  = p“(x(t) 

U A 

variation)  as  t -+  °°,  then  0 (u)  = 


t then+  y < 0(u). 

e D ) strongly  ( in 
k (x , u (x) ) u (dx) . 


If 


In  general,  we  do  not  know  whether  such  a smooth  V(*)  or  an 
optimal  control  exists.  Part  of  our  aim  is  to  replace  (1.5)  by  a 
local  maximum-principle  which  does  for  our  problem  what  the  work 
of  Davis  and  Varaiya  [ 1 ] or  Bismut  [ 2 ] did  for  the  control  problem 
on  a finite  time  interval  or  for  the  discounted  control  problem. 

A one  dimensional  version  - on  a finite  interval  with  reflection 
was  treated  by  Mandl  [ 5 ] , and  an  incomplete  development  of  the 
r-dimensional  version  of  Mandl 's  result  is  given  in  [ g]. 

In  Section  2,  the  solution  to  (1.1)  is  defined,  and  some 
properties  listed.  Sections  3,  4,  5,  6 deal  with  the  existence  of 
an  invariant  measure  for  each  u(*)»  with  certain  continuity 
properties  of  the  measure  with  respect  to  bu ( • ) and  with  the 


See  Kushner  [4]  for  a formal  discussion. 


existence  of  an  optimal  control,  with  the  existence  and  properties 
of  an  auxiliary  VC*)  function,  and  with  the  maximum  principle. 
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2 • The  Solution  to  (1.1) 

Let  C,  ^ and  ^ denote  the  space  of  Rr  valued  con- 
tinuous functions  on  (0,°°),  and  the  o-algebras  induced  by  the 

coordinate  projections  xCs),  s < t,  and  x(s),  s < °°,  resp.  , 

N N 

where  x(-)  is  the  generic  element  of  C.  Define  f - ( • ) : f (x)  = 

f (x)  if  | x | < N,  and  is  zero  otherwise.  Then,  for  each  x e Rr, 

there  is  a unique  measure  on  (C,&)  such  that  { , x e RN } solves 

the  martingale  problem  of  Strook  and  Varadhan  [ 3 ] . There  is  also 

x N N 

a standard  Wiener  process  W ' (•)  defined  on  (C,  j£,Q  ),  and 

adapted  to  the  (completed  with  respect  to  Q^)  { jg7  } and  such  that 

dx  ( t ) = fN(x(t))dt  + a (x  (t) )dwX,N  (t) , x(0)  = x,  w.p.l  Q ^ . 


Define  S,.  = (x:  |x|  < M}  . We  will  show  that  the  solution  is 

also  well  defined  for  N = <».  The  stability  condition  (A3)  yields 

the  following  result.  in  the  Lemma,  suppose  (w.l.o.g)  t^hat  is  in 

the  interior  of  S . 

M 

Lemma  2.1.  Assume  (Al)-(A2),  (A4).  Let  x f S..  and  N > M. 

- --  fl  

Define  o = inf{t:  x(t)  i SM  - K^},  o (K^  = inf{t:  x(t)  e f^}. 

Then  (PN,EN,  correspond  to  Q^) 


(2.1) 


P^{x(t)  hits  before  hitting  1 1 


(x) 

kM 


where  k = inf  W.  (x)  , 

M l 

I X | »M 

ExaM  - VX)/e 


(2.2) 


r 
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Proof . By  Ito's  Lemma,  for  any  t < °°  (_/°  corresponds  to  u(-) 

tn°M 

ExWi(x(t  naH)  ) = W1(x)  + eJJ  | ^°W1(x(s))ds 


N , 


< Wx(x)  - cE"(tnoM)  . 


This  inequality  implies  ExWl (x  ^ (x)  , Wx  (x)  > cE^,  from 

which  both  (2.1)  and  (2.2)  follow.  Q.E.D. 

Since  kM  -*■  °°  as  M -*•  °°,  it  can  be  shown  that  (2.1)  implies  that 

(2.3)  lim  sup  P^{  sup  |x(t)|  > M)  = 0,  for  each  x and  T. 

M-*oo  N X 

0<t<T 

By  virtue  of  (2.3),  there  is  a unique  solution  (P°  ) to  the 
martingale  problem  for  coefficients  (f,o),  and  each  x e R . 

Similarly,  since  neither  the  r.h.s  of  (1.3)  nor  depend  on 

u(-),  there  is  a unique  solution  (P^)  to  the  martingale  problem 
f°r  all  x e R and  coefficients  (f+b  ,cr),  where  u(»)  is  admissible. 
Furthermore , 

(2.4a)  EUa  (K.  ) < W-.(x)/e,  EU  corresponding  to  PU 

X X “ 1 X X 

(2.4b)  PU{sup  | x ( t ) | > N } -*•  0 as  N -*•  °°,  uniformly  in  u(-), 
x t<T 

and  in  x in  bounded  sets 

(2.4c)  P^{x(t)  hits  before  hitting  SS^}  > 1 - W^(x)/kN, 

if  SN  K1  and  x e SN* 

There  is  a Wiener  process  Wx,u(»)f  defined  on  {C , % , P^)  and 
adapted  to  {^fc}  (completed  with  respect  to  P^)  and  such  that 


(2.5) 


dx (t)  = [f (x (t) ) + bu(x(t))]dt  + o(x(t) )dWX,u(t) , w.p.l  (p") . 
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For  each  real  o < T ■<  <»,  define 


T 

^0(U)  = 


0 


[a  1 (x  (s)  } bu(x  (s)  ) ] 'dW*'  0 (s)  - \ [ | a-1  (x  (s)  ) bu  (x  (s)  ) ! 2ds 


0 


Then  (the  proof  of  (2.6)  is  the  same  as  that  of  Theorem  6.2  in  [3], 
where  f(-)  0;  see  also  Girsanov  [7]) 


(2.6) 


dP^  = exp  r,Q  (u)  *dP°  on  (C,5fT). 


Thus,  (at  least  on  each  (C,  50 ) , for  each  x all  the  measures  P 
are  mutually  absolutely  continuous,  so  that  a.s.  statements  with 
respect  to  one  are  also  a.s.  statements  with  respect  to  the  others 
on  each  (C,  ^T) • Also+  (a.s.  P°) 

(2.7)  dWx,0(t)  - a ^ (x  (t)  ) bU  (x  (t)  ) dt  = dWX,U(t). 

(See  Girsanov  [ 7 ] or  Davis  and  Varaiya  [ 1 ] ) 

By  [ 3 1 , the  solution  to  the  martingale  problem  with  co- 
efficients (fN,o)  or  (fN+bU,o),  for  admissible  bU ( • ) , is  a 
strong  Markov  and  a strong  Feller  process  and  in  each  of  these 
cases  the  measures  of  x(t)  have  densities  with  respect  to  Lebesgue 
measure  for  all  x = x(0)  and  all  t > 0.  Furthermore,  these 
densities  are  positive  almost  everywhere.  By  the  stability 
condition  (A4)  and  (2.3),  (2.4b),  these  facts  are  also  true  for 
the  solution  with  coefficients  (f+bU,o)  for  admissible  bU ( • ) . 
Define  P^{x(t)  e T}  3 Pu(x,t,D  and  denote  its  density  at  y by 
p (x, t , y)  . 


^Whenever  such  differentials  are  equated,  we  mean  to  equate  the 
corresponding  integrals. 


x c 
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The  following  Lemma  will  be  useful  in  the  following  sections, 


Lemma  2.2.  Assume  (Al)  - (A2)  and  (A4)  - (A5) . Define 


a(K2)  = inf  {t:  x(t)  e K2  U 1^}. 


Then 


..  7 2 [W  (x)  + c?W  (x) /c] 

(2.8)  E^a  (K2)  < — , x { K1  U K. 


ea 


'1  ~ 2 


Proof . By  Ito's  Lemma  and  (A5) 


t Oct  (K, 


0 < e^Jw2  (x(t  Ho  (K2)  ) ) < W2(x)  + ex 


[co-  «W1 (x (s) ) ]ds, 


from  which  we  get  (use  E^o(K2>  < °°) 


EUf 


o (K. 


0 < W2(x)  + Exj  [c2-aW^ (x  (s) ) ] ds, 


The  last  inequality,  (2.4a)  and  o(K2)  < o(K^)  imply  that 


0 < W2 (x)  + c2Exo (Kx)  -a  c 


0ExI (a (K2) >s }Ex (s) a ^K2  ^ ds' 


The  integrand  equals 


ExI{o(K2)>s} (a (K2]  ~ s) 


,u_  2 


Hence,  the  integral  equals  Exo  (K2)/2,  from  which  (2.8)  fol lows . Q. E . D . 
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3.  The  Invariant  Measure 


Let  G and  be  spheres  in  R , centered  at  the  origin, 

with  radii  y and  y1»  resp. , y < and  boundaries  r and  ri, 

resp. , and  with  G D U K2.  Define  x'  = inf{t:  x(t)  e F , } , 

= inf  { t : x(t)  c F},  x'  = inf{t:  t > x ^ , x(t)  e r^},  and  define 

and  x^,  n > 1,  recursively  by  x^  = inf{t:  t > x^  x(t)  e 

x'  = inf{t:  t > x , x(t)  e r.}.  x will  be  used  for  x_  - x.  = x_, 
n n i 2 1 2 

when  x c T.  Define  Xn  = x(xn>.  Then,  if  x c F,  {X^}  is  a (homogeneous) 
Markov  chain  on  the  state  space  F,  and  Khasminskii  [ 8 ] uses  it  to 
construct  the  invariant  measure  for  (x(t)}.  Let  x (A)  denote  the 

r T 

amount  of  time  ( Ir  />\ra)dt)  that  x(t)  spends  in  a Borel  set  A 

Jq  ■ X vl)  t A j 

during  [0,x2]  = [0,t],  when  x(0)  = x e r (if  x(0)  r,  then 

Theorem  3.1.  Assume  (Al)  - (A2),  (A4 ) . Then  there  is  a 

constant  c^ : 

(3.1)  sup  E^i  < c^  < °°. 

X£  F 


Both 

and 

X ( • ) 

each 

u ( - ) ) 

5« 

and 

(note 

that 

uu(R 

r)  = 

(3.2) 


PU(A)  = Uu(A)/uu(Rr)  , 


WU(A)  = j yu(dx)E"x(A)  . 


The  measure  y has  a density  (with  respect  to  Lebesgue  measure) 
which  is  positive  almost  everywhere  and  the  value  at  the  point  y 
is  given  by 
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(3.3) 


pU (x,t,y) uu (dx) 


For  any  bounded  Borel  function  F(*)/ 


(3.4) 


F(x)uu(dx)  = 


T'  U 


(dx) E„  F (x  (s)  ) ds. 


> 0 


Also 


(3.5) 


sup  Ex  < W (x)/e,  x £ G. 
n X 1 - 1 


For  each  Borel  set  A and  bounded  measurable  function  F(*)» 


(3.6)  PU(x,t,A)  - u (A),  E^F  (x  ( t ) ) •+  F(x)ji  (dx),  as  t - 

11  x I u 


Proof . Set  t q = infft:  x(t)  / G^}.  To  prove  (3.1),  we 
first  show  that,  for  fixed  t > 0 and  some  real  c < 1 


(3.7) 


inf  PU{ t l < t}  > 1 - c. 
x 0 - 

x , u 


(3.7)  follows  from  the  fact  that  there  is  a c < 1 such  that 


inf  P^{sup|x(s)|  > y.}  = inf  PU{sup|x  + 
x,u  s<t  x,u  x s<t 


(f (x (s)  ) + b (x  (s) ) ) ds 


a (x  (s)  ) dWx,U  (s)  | > Yl> 


> inf  P (sup | a (x (s) ) dw5*' u(s) | > Yl  + |x|  + Kt}  > 1 - c, 
u , xeG  s<t  Jo 
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where  K is  a bound  on  j f + b | in  . Now 


Px  r0  ntl  ExZ{t^>  (n-1)  t}I{i^)>nt} 


= E^I  EU  T < F^I  < 

itq>  (n— l)t}  x(nt-t)  (Tj>t}  - V(;b(n-l)t) 


which  implies  that  ^ nt  = c^  is  an  upper  bound  to  e^Tq. 

Ext ' 1 C4  f°r  x e f.  Indeed,  (to  be  used  later) 


Hence , 


(3.8) 


E^(T')a  < J(nt)acn_1  < x e G. 


Equation  (3.5)  follows  from  (2.4a),  since  G 3 Kj  UK0.  Thus,  for 


x e F , 


EUr  = EU  ' ds  + EU  ! ds 
x x JQ  x JT, 


= e“t'  + EUEU.  < c + EUW, (x(t' ) )/e  < c, 

x x x (x  ' ) 1 - 4 x 1 ' - 3 

for  some  real  c which  gives  (3.1). 

In  [8] , Khasminskii  proves  that  there  is  a unique  invariant 
measure  uu  under  the  conditions  (i ) : Pu  (x , t , A)  > 0,  all  open  A,  all 
x and  all  t > 0,  and  (ii):that  x(*)  be  recurrent  (Khasminskii ' s 
definition  of  recurrence  is  implied  by  (3.5))  and  (iii):  x(*)  is  a 
strong  Feller  and  a strong  Markov  process.  Under  the  additional 
condition  (3.1)  (for  fixed  u(*))  there  is  a unique  finite  invariant 
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measure  u given  by  (3.2)  ([8],  Theorems  2.1,  3.2  and  3.3). 

Equations  (3.3)  and  (3.4)  follow  by  simple  calculations. 

Since  uu  and  Pu(x,t,-)  all  have  densities  which  are 
positive  almost  everywhere  (all  x,  and  all  t > 0) , they  are 
mutually  absolutely  continuous.  Equation  (3.6)  then  follows  from 
[8],  Theorem  3.4  or  Doob  [8],  Theorem  5 (let  his  4>  equal  our  ^ 
and  his  P.^  = 0),  since  uu  and  Pu(x,t,*)  are  mutually  absolutely 


continuous  for 


t > 0.  Q.E.D. 
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4 . Existence  of  an  Optimal  Control 

Theorems  4.1  and  4.2  provide  some  preliminary  results.  Theorem 
4.3  proves  that,  in  a sense,  the  invariant  measure  is  continuous 
in  the  function  b (•) . This  leads  directly  to  the  existence 
Theorem  4.4.  Define  g(*,*)  = (b  k (•,*)) . 

Theorem  4.1.  Assume  (Al)  to  (A3).  Let  {un(-)}  denote  a 

u u 

sequence  of  admissible  controls,  and  write  g (')  = (b  n(-),k  n(.)). 

If  there  is  a bounded  measurable  function  g(*)  such  that 

gn(x)dx  g(x)dx,  all  Borel  A, 

J A > A 

then  g ( • ) is  admissible  in  the  sense  that  there  is  an  admissible 
u(*)  such  that  g C * ) = ( bu  (*),ku(*)),  for  almost  all  x. 

Proof.  The  theorem  is  a standard  existence  theorem.  See 
Roxin  [10]  or  McShane  and  Warfield  [11],  By  an  argument  such  as  that 
used  by  Roxin  [10],  g(x)  e {y:  y = g(x,a),  for  some  a e ^ } = g(x,^) 
for  almost  all  x.  We  can  assume  that  the  conclusion  holds  for 
all  x.  Then  the  theorem  follows  by  the  implicit  function  theorem 
in  [11].  Q.E.D. 

A family  {<]>  } of  measures  on  Rr  is  said  to  be  tight  if 
for  each  c > 0,  there  is  a compact  such  that  4>ci(Rr-K[_)  < e, 

l(’)  denote  Lebesgue  measure. 


all  a.  Let 
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Theorem  4.2.  Assume  CAl)  - (A2)  , (A4)  - (A5)  . Then  {y^ , 
u(  •)  admissible]  is  tight.  Also,  yu(A)  -*■  0 as^  l (A)  0,  uniformly 

in  u (•  ) , A. 


Remark.  The  theorem  is  true,  but  harder  to  prove,  without 
(A5).  Since  (A5)  will  be  used  later  anyway,  we  use  it  now  to 
simplify  the  proof. 

— 2T 

Proof . Since  PUCR  ) is  bounded  uniformly  in  u(*)  (by 

virtue  of  (3.1))  to  show  tightness,  we  only  need  to  show  that  (refer  to 
the  definition  of  y^  in  (3.2)) 


sup  EUx  (S')  ->0  as  N -*■  °°, 
xeT  x N 
u 


where  S^  = { y : |y 

{y:  | y | < N]  3)  G ^ 

Then, for  x e r , 


> N}.  Recall  that 
(See  (A4 ) , (A5 ) for 


Gj  D U , and  assume  that 
the  definition  of  K^K^.) 


T(S'l 


(t  (S^) >0} 


2 


< sup  E^m-sup  P^{  T (S')  > 0}  E M.M,. 
xcT1  x 1 xcT  X N 1 1 

u u 


By  (2.4c),  M~  < sup  W.  (x)/k  and  by  Lemma  2.2,  M.  < °°.  The  first 
xel^  1 

assertion  of  the  theorem  now  follows,  since  k -+  » as  N -*■  «>. 

N 

Fix  e and  (by  tightness)  choose  compact  such  that 

UU(KC)  > 1 - e,  all  u(- ) . Then,  for  t > 0, 
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Uu(A)  = yjuCdx)  pU(.x,t,A)  + 

Rr-K 

e 


< e + sup  F u (x , t , A) . 
xcK  _ 


K 


yuCdx)P  (x , t , A) 


Also,  by  (2.6)  (recall  that  the  superscript  o corresponds  to 
u = b = 0)  , 

.u,._  . ...  2 


[P  (x  , t , A)  ] = [E°I{x(t)eA}exp  ^(  u)]2 


- ExJ{x(t) eA}Ex  exp  2^0(u  ] - constant • P° (x , t , A) 


The  last  assertion  of  the  theorem  follows  from  the  last  two  inequalities 
since  is  arbitrary  and  P°(x,t,A)  -*■  0 uniformly  in  x c K and 

in  A,  as  1(A)  -*  0.  Q.E.D. 


Theorem  4.3.  Assume  (Al)  to  (A5) . Let  b(-)  be  a bounded 

u 

measurable  function,  and  { b n ( • ) } admissible,  such  that  (write  sub 
or  superscript  n for  un ) 

bn(x)dx  [ b (x)  dx , all  Borel  A. 

A J A 

Then  (Theorem  4.1)  there  is  an  admissible  u(*)  such  that  b(-)  = 
bU  ( • ) a.e.  Also, 

(4’2)  exp  cjj(un)  - exp  cj(u) 

weakly  in  (with  respect  to  P°)  as  n - for  each  x and 

t > 0.  In  particular, 
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(4.3) 


Pn(x,t,A)  •>  PU(x,t,A) 


(4.4)  E^F(x(t))  -*  E^F  (x  (t)  ) , each  x,  t > 0,  Borel  A,  bounded 

measurable  F(*). 


(4.5) 


Un(dx)F(x)  -*■  yu(dx)F(x),  F ( • ) bounded  and  measurable. 


Proof.  The  convergence  (4.2)  is  proved  by+  Bismut  [2  ], 

Theorem  IV-3,  and  (4.3)  , (4.4)  follow  from  that  convergence  and  (2.6). 

Since  {yn)  is  tight,  it  is  weakly  sequentially  compact, 
(Billingsley  [12]/  P-  37).  I.e.,  each  subsequence  contains  a 

further  subsequence  { v;  n } such  that,  for  some  probability  measure  ;j , 

l 

F(x)yn  (dx)  -»■  F(x)y(dx),  all  bounded  continuous  F(*) 
i 


(Billingsley  [12],  pp.  35-37).  Let  n index  such  a convergent  sub- 

A 

sequence,  with  (weak)  limit  y. 

Let  F ( • ) be  bounded  and  continuous.  Let  e > 0 and  define 
K as  in  Theorem  4.2  and  write 


Un(dx)F(x)  = 


yn (dx) ExF (x (t) ) 


y (dx)  E*  F (x  (t)  ) + 

K rJ 

e R -K 


Un  (dx) ExF (x (t) ) . 


undereach  control, 

The  second  term  is  < e sup|F(x)|.  Since  x(-)  is  a Feller  process 

x 

ExF(x(t))  is  continuous  in  x,  for  each  v(-).  Then,  the  function 


+In  [ 2 ] , f = 0,  but  the  proof  is  exactly  the  same. 


f 
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E^F(.xCt))  is  continuous  in  x and  converges  to  the  continuous 
function  E^F(x(t)),  by  (4.4).  This  convergence  implies  that, 
for  each  6 > 0,  there  is  a Borel  set  C K , with  £(Af)  < 6, 

- 0 E 0 - 

such  that  |EUF(x(t))  - E^F(x(t))|  < e for  large  n,  and  x { A,,  xeK  . 
Those  estimates,  Theorem  4.2,  the  arbitrariness  of  6 and  the 
weak  convergence  imply  that  the  first  term  on  the  right  goes  to 

I ^ u 

u(dx)E  F(x(t)),  as  n -+  °°.  Since  the  l.h.s.  converges  to 

i K x 

e 

j u(dx)F(x),  and  e > 0 is  arbitrary,  we  conclude  that 


J 


U (dx)  F (x) 


U (dx)E^F(x(t)  ) . 


This  equation  together  with  the  arbitrariness  of  t > 0 and  F(-), 

implies  that  y is  an  invariant  measure  - under  control  u(*).  Thus , 

the  uniqueness  Theorem  3.1  implies  that  u = P . Since  the  result 

does  not  depend  on  the  selected  subsequence,  we  have  that  un  -*  Uu 

weakly,  as  n -*•  °°,  and  (4.5)  holds  for  bounded  and  continuous  F(-). 

Let  F ( • ) be  bounded  and  measurable.  Then,  for  t > 0,  the 

invariance  of  y implies 
n 


Un (dx) F (x) 


Un(dx)Fj(x) , Fj(x)  = E^F (x (t) ) . 


By  the  strong  Feller  property,  E^F(x(t))  is  continuous  in  x. 
Now,  as  in  the  proof  of  (4.5)  for  continuous  F(*)f  the  almost 
uniform  convergence  of  F™(x)  to  E^  F(x(t)),  Theorem  4.2  and 
the  tightness  and  weak  convergence  of  {y^}  imply 
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PR  Cdx)  Ffc  (x) 


p uCdx)  ExF  Cx  Ct)  ) , 


which  must  also  equal  the  limit  of  p (dx)F(x).  This  implies 


(4.5),  since  by  the  invariance  of  pu  , the  r.h.s.  equals 


p (dx)F(x).  Q.E.D. 

J u 


Theorem  4.4.  Assume  (Al)  to  (A5) . Then  there  is  an  optimal 
admissible  control. 


Proof . Let  (u^C*))  denote  a minimizing  sequence.  Then 

0 E lim  0 (u  ) = inf  0 (u) . Let  n also  index  a weak  star  (a (L  ,L.) 

u(.)  00  1 

topology) convergent  subsequence  of  (bn  ( • ) , kn  ( • ) } with  limit 
(b  ( • ) » k ( • )),  where  we  let  n replace  the  index  u^.  There  is  an 
admissible  control  u(*)  such  that  (Theorem  4.1)  (b  (*),k(*))  = 

( bU  ( • ) , ku ( • ) ) . If  k(*)  does  not  depend  on  the  control,  then 
0 (un)  -*■  0 (u)  by  (4.5).  Hence,  in  this  case  there  is  an  optimal 

control . 

Now  let  k(*)  depend  on  the  control.  Let  Fn  ( • ) be  a 
sequence  of  bounded  measurable  functions  which  converges  to  a function 
F ( • ) in  the  weak  star  topology.  Then+  (Bismut  [2],  Proposition  IV-4, 
p.  48) 


F (x(s))ds  •>  F(x(s))ds 

0 n J 0 


in  probability  (P°,  each  x)  , as  n Note  that  E°  exp  2^^(v) 

is  bounded  uniformly  in  x and  in  the  control  v(*).  Let  Fn  ( • ) 


In  [ 2 ] , f = 0,  but  the  proof  is  exactly  the  same  :or  our  case. 
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be  defined  by  Fn(x)  = k(x,u  Cx)),  and  set  F(_x)  = k(x,u(x)). 

By  the  convergence  in  probability,  the  convergence  (4.2)  and  the 

boundedness  of  E°exp  2 r;^(u  ) uniformly  in  n,x,  we  have 

x u n 


(4.6) 


^ J0FnU(s))ds  = ExexP  ^o(Un) 


•t 

F (x (s) ) ds 
„ n 


^o  t , . 

-*•  Ex  exp  r>0  (u) 


■t 

F (x (s) ) ds 

0 


E 


u 

X 


F(x(s))ds  (a  continuous 

0 

function  of  x) . 


Integrating  the  left  and  right  sides,  resp.,  of  (4.6)  with  respect 
to  and  yu,  resp.,  and  using  the  invariance  of  these  measures, 
yields  the  two  equations 


(4.7a) 

(4.7b) 


t0 (u  ) = 
n 


t u 

yn(dx)  | E^r‘k  (x  (s)  , u_  (x  (s)  ) ) ds 


t0 (u)  = 


V (dx ) | E k (x  (s)  , u (x  (s)  ) ) ds . 
0 x 


Now,  (4.6)  implies  that  the  right  hand  integral  in  (4.7a)  converges 
to  that  in  (4.7b)  for  each  x.  This,  together  with  the  tightness  of 
(unl,  the  last  part  of  Theorem  4.2,  and  an  argument  like  that  used 
in  the  proof  of  Theorem  4.3  to  show  (4.5)  yields  that 


0 (u  ) -*  0 (u)  . 
n 


Q • E . D . 
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5 • The  Auxiliary  yU  ( • ) Function 


Our  aim  is  to  get  a replacement  for  the  V(-)  function  in 
(1.5),  which  will  play  an  important  role  in  the  sequel. 

In  this  section  the  control  u(>)  is  fixed,  and  we  return 
to  the  Markov  chain  {X^}  °f  Section  3.  For  a measurable  set 
y e T , let  tt  (x,y)  = PU{Xn  e y},  x e T,  and  recall  that  the  unique 

LI  XX 

invariant  measure  for  fX  } is  denoted  by  jj  . Let  6 be  a 

n u 

finite  measure  on  r.  The  chain  {X  } is  said  to  be  uniformlv  d> 

n *- 

recurrent  if  for  each  measurable  y e T such  that  <J>(y)  > 0, 
n 

(5.1)  ) P (X  e y , X • i Y r i < m}  -*■  1 uniformly  in  x e F, 

i x m l 

m=l 

as  n 

(Orey  [13],  p.  26).  A sufficient  condition  for  (5.1)  is  (Orey  [13], 
p.  29)  that  if  $(y)  > 0 then  there  is  an  n < «=  and  e > 0 (perhaps 
depending  on  y)  such  that 

n 

(5.2)  7 PU { X e y,  X.  £ y , i < ml  > e 

_ i x m i T 

m=l 

for  all  x e T.  If  the  chain  is  uniformly  <)>  recurrent  and  a-periodic 
then  there  are  constants  c and  p e (0,1)  such  that 


(5.3)  lPx^Xn  c i cpD 

uniformly  in  y and  x e T (a  consequence  of  equation  (6.2)  in 
[13],  p.  26,  and  the  invariance  of  u ) . Thus,  the  n-step 
transition  probability  7T^n)(x,-)  converges  to  uu  in  variation, 
at  an  exponential  rate. 

Define,  for  x e Rr  (see  Section  3 for  the  definition  of 
and  pu)  , 

_ . _ ~ » ..  . . . . . — . ^ 
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(5.4) 


VU(x)  = EU 
x 


1 


[k  Cx  (s)  ) - 0 (u)  J ds 


(5.5) 


n 


+ 1 im  y [ E 


u 


n m=l 
x. 


m+1 


J x 


ku(x(s))ds  - | uu(dx)ku(x) 


m 


x 

III— X 

Vu(x)  = E^  ! xu(x(s))ds  + lim  E^  [ ku(x(s))ds, 


* J 


0 


1 


where  k (x)  = k(x,u(x))  - 6 (u)  . 


Theorem  5,1.  Assume  (Al)  - (A2),  (A4 ) - (A5 ) . Then  V (x) 


and  VU(x)  are  well  defined.  There  are  constants  c0'Ci  such  that 


(5.6) 


iVU(x)|  < CQ  + C^x, 


(VU(x) | < CQ  + C1E^r1. 


The  tail  of  (5.5)  (E^_  j • ) goes  to  zero  as  n,m  uniformly  in 


x,  and  E 


x J 

'~u  n 

k (x(s))ds  is  bounded  uniformly  in  n and  x e F. 


Proof . Let  cj>  = Z = Lebesgue  measure  on  T.  n ( x ,y)  >0  if 
y is  open  in  T.  Then  ttu(x,y)  > 0 if  My)  > 0.  Since  ttu(',y) 
is  continuous  (by  the  strong  Feller  property  - it  also  follows 

from  the  assertion  6°  of  Khasminski  in  [8],  with  a suitable  definition 

of  U,r  there),  inf  tt  „ (x,y)  > 0.  Thus,  by  the  criterion  (5.2), 

xcr  u 

with  n=  1,  {X  } is  uniformly  ^.-recurrent . 
n 

A 

Let  F ( • ) be  a bounded  measurable  function  on  F.  By  virtue 
of  (5.3), 


1 


I 
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(5.7)  £ | E^F  (X  ) - u (dx)F(x)|  5 constant,  x e F. 

x n t-i 


Let  F (x)  = E' 


Note  that 


k (x(s) )ds,  which  is  bounded  on  T by  (3.1) 


0 


u ~ 

E F(X  ) 
x n 


EUEU 
X X 


: r n+1 

kJ  (x  (s)  )ds  = E I kU(x(s))ds. 

) X T 

n 


Then,  by  using  (3.4)  and  (5.7), 


00  f n+l  f _ 

(5.8)  [ | E^  j kU(x(s))ds  - Mu (dx) kU (x) (Rr) [ < constant,  x c F, 


n=l  1 t ' 

n 

rTl 

EU  [kU 


n 

This,  together  with 


(x(s))|ds  < sup | ku Cx)  j E^t^  , implies  both 


that  V (•)  is  well  defined  and  also  the  first  line  of  (5.6). 

^ u 

Now,  redo  the  above  argument  with  F(x)  = E^t,  x g F.  Then 


(5.9) 


y ieueu 

n=0  X X 


u (dx)Eil  < constant,  x - F 
, u x 1 - ' 


n 


Noting  that 


Pu(dx)Exx  = pu (R  ),  the  convergence  in  (5.8)  and 


r ' n+1 

(5.9)  allows  us  to  replace  u (Rr)  in  (5.8)  by  E^EU  t = EU  ! ds, 

U X ~ X J 

n 1 n 

and  still  to  get  convergence.  From  this,  we  get  both  that  VU ( • ) 
is  well  defined,  and  the  last  bound  in  (5.6).  The  last  two 
assertions  of  the  theorem  follow  from  (5.8)  and  (5.9).  Q.E.D. 


The  Lemma  gives  some  useful  estimates. 


See  (3.4),  with  F(x) 


1. 
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The  constant  C may  have  different  values  in  each  usaqe. 


Lemma  5.1.  Assume  (Al)  - (A2) , (A4)  - (A5) . Then,  for  all 
admissible  u(*),v(*)  and  all  x,  s > 0,  t > 0,  and  Markov  times  a < t. 


(5.10)  E^! Vu Cx (t) ) : < C [ 1+W^ (x) +t] 


(5.11a)  EV  , . | VU  (x  (t+s)  ) | < C[l+W1U(t))+s] 


(5.11b)  EV  1 VU  (x  ( (s  + t)fl  p ) ) | < Ctl+W1(x(t))+s],  w.p.l  for 


x ( t ) - 

any  Markov  time  P > t 


(5.11c) 


E,.  ,W.  (x(t  + s))  < C [ 1+W  . (x  (t)  ) + s]  w.p.l. 

X ^ L.  ) 1 “ 1 


(5.12) 


E>U(x(B))|  < C [ l + t+W0  (x)  ] , each  u ( • ) , v ( • ) 8 = a Pi  t 


:s. i3) 


«v  (dx)  [ Vu  (x)  < °°,  j uv(dx)  jW^(x)  I < each  u,v. 


Proof.  By  (5.6), 


E^jVU(x(t))|  < constant [1+E^E^ (t) ij] 


By  (A4),  there  is  a constant  such  that  1/  W^(x)  < e^,  all  x 

and  v(-).  Thus,  by  an  application  of  Ito's  Lemma, 


ELWi  (x(t)  ) < W^'x)  + e^. 


The  last  two  equations  and  the  bounds  (3.5)  and  (3.8)  imply  (5.10) 
Equation  (5.11)  follow/  by  similar  calculations  . 
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We  prove  (5.12)  only  for  a = t.  The  general  proof  is 
similar.  Write 

VU  (x (t) ) < C[l+E“(t)Tl]  < C[l+W1(x(t) ) ] . 
Continuing,  and  using  (5.11c)  and  (A5), 


E^|vU(x(t) ) |2  < CE^[1+W2 (x (t) ) ] 


C[l+W2 ( x ) + t ] . 

To  prove  (5.13),  define  aN  = inf ft:  |x(t)|  > N},  and  for 
integer  M define  l^(x)  = min  [Wx  (x)  ,M]  , and  note  that 

(by  (A5)  and  Ito's  Lemma) 


t O a 


E>!ll,inVl  i Vx>  + j V2-q2<*(s))]dS 


Thus,  by  bounding  q_ ( ■ ) and  letting 


N -*■  00 , 


0 < VI  (x)  + EV 
- 2 x 


[c2~aW^ (x (s) ) ]ds. 


Divide  the  lust  equation  by  t,  let  t -*•  and  get  (using  (3. 
convergence  of  PV(x,t,«)  to  the  invariant  measure  uv(*)) 

c > lim  i j oeV-Vx  (s)  )ds  = a [ y (dx)W^(x). 
t-*-®  J 0 x 1 J v 1 


each 


6 ) , the 


-26- 


Hence,  c2  > a j ^(dxjW^x).  This,  together  with  VU(x) 

C-  [ 1+Wj^  (x)  ] , implies  (5.13).  Q.E.D. 

Theorem  5.2  will  be  used  to  obtain  the  optimality  criterion  in 
Theorem  6.1. 


Theorem  5.2.  Assume  (Al)  - (A2)  and  (A4)  - (A5) . Then 
VU  ( • ) is  continuous  and  the  process  M^.  given  bv 


m“  = VU(x(t) ) - VU(x)  + 


k (x  (s) ) ds 


is  a continuous  square  integrable  martingale,  adapted  to  { ) and 

under  P^,  each  x. 


Proof.  First  we  note  several  facts.  Vu ( • ) does  not  depend 
on  . If  Y < Y'  < Yjy  and  GJ  is  a sphere  with  radius  y ' , 
then  several  cycles  (T  -+  F 1 , etc.)  of  the  process  for  the  G|  case 
may  be  included  in  one  cycle  for  the  G1  case,  but  the  values  of 

sX 

V (•)  are  the  same.  Also,  as  Y-.  Yr  sup  E^x  0.  Note  also  that 

xeT  x 

inf  P{t  - T}  -*•  1,  as  n -+  °°,  each  T < °°. 

v * 11 


Let  n denote  the  largest  integer  i such  that  x.  < T. 


l - 


Write  k (x(s))  as  k(s).  It  will  be  shown  that 


(5.14) 


u f m 

lim  lim  E k(s)ds  = 0,  uniformly  in  bounded  x sets. 

T-*-°°  m,n-»oo  J t Hi 
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This  will  imply  that 


VU(x)  = eJJ 


rT 


fc  (s)  ds  + e (T,x)  , 


where  e(T,x)  -*•  0 as  T -*•  »,  uniformly  in  bounded  x sets.  This 
and  the  strong  Feller  property  imply  that  VU ( • ) is  continuous. 
Now,  for  some  constants  C^,  and  any  integer  Q, 


lim  E' 
m,  n 


m 


t n t 


k ( s ) ds 


: E° 
x 


Nt+1 


k (s)  ds 


1 

+ EU  l 

X i >nm  r. 


ri+l 


k (s)  ds 


± ClExl  V1 


lj+  l Eu  f 

NT  itn  xlJ 


T i+1. 


1<Q 


k ( s)  ds  j + C2E^tqI{q^}. 


The  first  and  third  terms  on  the  right  can  be  made  arbitrarily 
small  (uniformly  in  bounded  x sets)  by  selecting  large  T and 
small  (y1-y) , and  large  T,  resp.  The  central  term  can  be  made 
small,  uniformly  in  x,  by  choosing  Q large.  This  implies  (5.14! 

It  can  be  shown  that 


Ex(t) tvU(x(t+s>)  " VU(x(t))  + 


s+t 


~u 


k (x(v))dv]  = 0,  all  s > 0,  t > 0, 


where  the  conditional  expectation  above  exists  by  Lemma  5.1.  The 
martingale  property  follows  from  this.  The  continuity  and  square 
integrability  follow  from  the  continuity  of  VU ( • ) and  (5.12), 
resp.  Q.E.D. 
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6.  The  Maximum  Principle 


Let  u(*)  be  admissible.  It  will  be  seen  in  Theorem  6.2 
that  there  is  a Borel  function  (a  r-row  vector)  i|jU  ( • ) such  that 

for  each  x e R , 


(6.1) 


M. 


1|^U  (x  (s)  ) o (x  (s)  ) dwyi'u  (s)  w.p.l  P^, 
0 X 


E 


u 

X 


ft 

0 


| i^U  (x  (s)  ) 


| 2ds  < 


00 . 


Let  v(*)  be  admissible.  Then  x(*)  satisfies  (w.p.l  P^) 


dx  (t)  = [ f (x  ( t ) ) + bv(x(t))]dt  + a(x(t))dWx'v 


By  (2,7),  we  can  suppose  that  (w.p.l  P^) 

(6.2)  dWx,v(t)  = dWX ' 0 ( t ) - c-1(x(t) )bV(x(t) )dt 

= dWx,u(t)  + a-1  (x  (t) ) (bu (x (t) ) - bv(x(t)))dt. 


Since  P°,PU  and  Pv  are  mutually  absolutely  continuous,  all  a.s. 

XX  X 

statements  with  respect  to  one  are  also  a.s.  statements  with  respect 
to  the  others. 

Theorem  6.1  is  the  "maximum"  or  "Hamilton-Jacobi"  principle, 
a natural  development  for  our  problem,  of  some  of  the  ideas  in  [1]  and  [2). 

Theorem  6.1.  Assume  (Al)  - (A2),  (A4)  - (A5) , and  let  u(*)» 
v(-)  be  admissible.  If 
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(6.3)  eU,vCx)  H (kU (x)  - kv (x)  ) + ipu  (x)  (bu  (x)  -bv  (x) ) > 0 


on  a set  A of  positive  Lebesgue  measure,  then  there  is  an 
admissible  control  v (• ) such  that  6 (v)  < 0 (u) . The  condition 
eu'v(x)  < 0 a.e.  for  each  admissible  v(-)  is  necessary  and 
sufficient  for  u(*)  to  be  optimal. 


Proof.  First,  we  derive  the  basic  formula  (6.5).  Using  (6.1),  (6.2) 

and  the  definition  of  Mu  yields  (a.s.  P“) 

t A 


f t ^ 

0 = VU(x(t))  - VU(x)  + ku(x(s))ds 

J 0 


4jU(x(s))o(x(s))  [dVVx,v(s)  - a'1  (x(s)  ) (bU(x(s)  ) - bV  (x  (s)  ) ) ds] 


Define 


a 


N 


min{ t : 


/t  2 

| i)u  (x  (s)  ) a (x  (s)  ) 1 ds  = N}. 
J 0 


Then 


(6.4) 


tna 

f 

0 = EV(x(tnoM)  ) - Vu(x)  + ; kU  (x  (s)  ) ds 

X N x J o 


+ E. 


ftrU,N  , 

»r  (x(s) ) [bU (x  (s) ) 

0 


bv  (x  (s)  ) ] ds 


where  the  expectations  exist  by  Lemma  5.1. 

By  the  uniform  integrability  implied  by  (5.12),  the  first 
term  on  the  r.h.s  of  (6.4)  tends  to  E^VU  (x  (t)  ) as  N -*  ®.  Also,  by 

in  Sq(v-u)  not  Wx'°), 


(2.6)  (use  W' 
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|^UCxCs))|ds  = exp  )* 

tnaN  tn°N 


I lJjU  (x  (s)  ) I ds 


ft  1/2 

< [E^  exp  2 cjj  ( v-u.  ,1/2[e“(  UU(x(s)  )|  ds)2]  H AjA^. 


tHo 


N 


A^  < *> , and  A2n  -*■  0 as  N -*•  ® by  (6.1).  Thus,  we  can  replace 
t D a by  t throughout  (6.4).  Setting  a = 0 in  the  above 

IN  JJ 

equation  yields 


| (x  (s)  ) | ds  < constant  [E^  | \pu  (x  (s)  ) | ds] 


2,  1/2 


But 


E 1 1/'  (x(s))|  < constant  f E^  ! Vu  (x  ( t ) ) - Vu(x)'2  + 1 + t2  : 

J 0 " x 


< constantfl  + t + W^(x)}‘ 


Then  by  the  last  inequality,  Swartz's  inequality,  (5.11c),  (A5)  and 

(5.13),  E^  j ip  (x(s)  ) [b  (x(s))  - bv  ( x ( s ) ) ] ds  is  in  t eg  r able  with 
respect  to  j.  v . 


Furthermore,  with  oN  H t set  equal  to  t,  the  Ex  can  be  put 


inside  all  the  integral  signs  in  (6.4).  ooinq  tMr  an^ 
each  term  with  respect  to  u , and  using  the  invariance  oi  u 


1 1 na 


(under  control  v(«))»  yields 


Mv (dx) [ExVu (x (t) ) - Vu(x)]  = 0 and 


O = 


ds 


(dx)  Ex(ku  (x  (s))  + i^u  (x  (s)  ) (bU  (x  (s)  ) - bV  (x  (s)  ) ] }. 


Now  subtract  the  zero  quantity 


Uv(dx)k  ( (x) ) from  the  above 


equation,  and  use  the  invariance  of  u (under  control  v(*))  to  get 
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O 


yyWx)  [ (ku  (x ) 


kv  (x)  ) + i^U  ( x ) (bU  (x) 


v 1 
b (x) ) r 


r 


or,  equivalently 


(6.5)  0 


My  (dx)  [eU,V(x)  + 0 (v)  - 0(u)]. 


Next,  let  A = {x:  eU'V(x)  > 0)  and  let  H (A)  > 0.  Define 
the  admissible  control  v(*)  by:  v(x)  = u(x)  on  Rr  - A,  v(x)  = 
v(x)  on  A.  Since  (6.5)  holds  for  all  u(*)»v(«),  eu'^(x)  > 0 


O 


U_(dx) [eU/ V (x)  + 6 (v)  - 0 (u) ] . 
v 


But  eu,v  (x)  >0  on  A,  and  u_(A)  >0  by  Theorem  3.1.  Thus, 

v 

0 (v)  < 0 (u) , proving  the  first  assertion  of  the  theorem.  The 
second  assertion  follows  easily  by  the  same  type  of  argument  on 
(6.5).  Q.E.D. 


Remark . The  reason  for  inserting  the  corollary  is  discussed 
after  the  proof. 


Corollary.  Assume  (Al)  to  (A5) . Let  u(*)  be  optimal  and 
v ( • ) an  admissible  control.  Then 


(6.6) 


0(v)  = 0(u) 


Uv (dx)eU'V(x)  . 


and 


Let  i|iU  ( • ) be  bounded  on  bounded  x-sets.  For  each  e > 0,  let 


::  ( • ) denote  a (r-row  vector)  Borel  function  such  that 


(6.7) 


1 1|'  _ (x)  - i{/U  (x ) | dx  -*■ 
• K ' 


sup  lip  (x)  I < 0°, 

£ , x e:  K ' 


0,  each  bounded  set 


each  bounded  set  K. 


K , as 


e 


0, 


If 

Let  K denote  a fixed  compact  set  in  R , which  is  the  closure  of 
its  interior.  Suppose  that  the  function  vc ( • ) is  calculated  by 


(6.8)  v (x) 

C 


v.  (x) 


arg  inf  [k(x,a)  + f (x)b(x,a)], 

a 

0,  x $ K. 


for  almost 

all  x e K , 


Then  v ( • ) 
e 


can  be  assumed  to  be  admissible,  and 


(6.9) 


lim  9 (v  ) < 6 (u) 
£ *■  0 


1 im 
“TIT 


(dx)  [ (ku(x) 


k°  (x)  ) + i|iu  (x)  bu  (x)  ] . 


Proof . Equation  (6.6)  is  just  (6.5).  Bv  the  complete  lattice 
property  ([14],  p.  302)  of  L^(K),  the  inf  in  (6.8)  can  be  assumed 
to  be  in  L^(K).  Then,  by  the  properties  of  b ( • ) , k ( • ) in  (A2)  and 
(A3) , we  get  that  the  inf  (evaluated  at  x)  is  in  the  set 

k(x/#)  + ip  (x)b(x,^),  for  almost  all  x e K.  Then,  the  implicit  function 
theorem  cited  in  Theorem  4.1  can  be  used  to  show  that  there  is  an 
admissible  control  which  attains  the  inf  almost  everywhere.  We  call 

this  control  v (•). 

c 
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Note  that,  if  Yc(x)  -*■  (x)  for  a fixed  x e 

(6.10)  inf  [k  (x,«)  + <p  (x)b(x,a)]  + inf  [k(x,a) 

a c% 


K,  then 

+ tpU  (x)  b (x  , at)  ] , 
as  c -*•  0. 


Also,  the  convergence  (6.7)  implies  that  for  each  .S  0,  there  is  at 

c t > 0 and  a set  A s e K with  MA^)  < 5 for  • > ^ and  such 

that  |tpr  (x)  - tpU  (x)  | < <5  for  x|  A^,  c < £5.  This,  together 

with  (6.10),  implies  that  the  difference  between  the  sides  in  (6.10) 
converges  in  L^K)  as  e - 0.  Note  that  the  r.h.s.  of  (6.10) 
equals  ku(x)  +1-  U(x)bU(x)  (almost  everywhere)  by  optimality  of  uC', 

and  the  theorem.  Now,  by  (6.6), 


0 (ve) 
(6.11) 


0 (u) 


v . v 

11  (dx)  [ (ku  (x) -k  ! (x)  ) + vU(x)  (b  u(x ) -b  (x)  ) ] 

Jk  v> 


r 

rJ  V r 
R -K 


(dx)  [ (ku  (x)  -k°  (x)  ) 


+ ipU  (x)  bU  (x)  ] . 


The  integrand  of  the  first  integral  of  (6.11)  equals 

v v.  u Ve 

[ku  (x)  + ipU  (x)  bu  (x)  ] - [k  ' (x)  + 4'£(x)b  (x)  ] - (ip  (x)  - ip£(x))b  (x). 

The  remarks  below  (6.10),  , . . . ,,,  _ •_  „/« 

/ and  the  fact  that  uv  (A)  -*0  as  £ (A)  - 0 uniformly  in  \ ( ) 


and 


(Theorem  4.2))  imply  both  that  the  first  integral  on  the  r.h.s. 


of  (6.11)  goes  to  zero,  as  e 0,  and  the  theorem.  Q.E.D. 
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W 


Remark  on  the  Corollary.  The  corollary  was  given  because  it 
will  probably  be  useful  when  used  in  conjunction  with  a procedure 
for  computing  or  estimating  ipU  ( - ) - Usually,  we  would  not  be  able 
to  calculate  exactly,  and  the  corollary  asserts  that,  even 

if  the  computation  is  approximate,  its  use  to  get  a control  may 
yield  good  results,  since  the  cost  is  "continuous  in  ( • ) " , in  a 
sense,  provided  that  j uv  (dx)  | ij>u  (x)  | -+  0 as  N 00 , uniformly  in 


v(*).  We  would  expect  that  this  latter  condition  would  hold  quite 

often . 

Theorem  6.2.  Assume  (Al)  - (A2),  (A4)  - (A5) . Then  M^  has 

the  representation  (6.1). 

Proof . By  Theorem  2.3  of  Davis  and  Variaya  [1],  and  the  square 
integrability  of  M^,  there  is  a process  £u,x(-)  such  that 

EU  1 | r. U ' X ( S ) | ds  < 00  for  each  t and  such  that 

X J0 

rt 

Mu  (s  ) = I CX  ' U (s)  a (x  ( s)  ) dWx ' u (s)  , w.p.l.  PU. 

J o x 

Let  Q^u  denote  the  class  of  continuous  random  functions  that  are 
square  integrable  martingales  under  P^,  each  x,  and  are  also  homogeneous 
additive  functions  of  the  Markov  process  x(*)r  and  which  are  adapted 
to  f ^ } . If  N ( • ) g -&U,  then  the  quadratic  variation  <N,N>t  has 
a representation  which  is  a homogeneous  additive  non-decreasing  function 
of  x(*).  It  does  not  otherwise  depend  on  x(0)  = x.  (See,  for 
example,  Mayer  [151,  Theorem  3,  p.  126.  The  result  is  also  implied 
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by  Kunita  and  Watanabe  [16],  Appendix.)  The  processes  MU ( • ) 
and 


*•  i 

a (x(s))[dx(s)  - (f(x(s))  + bu  (x  (s)  ))]ds] 

0 


are  in  u/u.  Also  WX,U(*)  ± MU  ( • ) are  both  in  Then 


<WX,U  + MU,  WX,U  + MU>t  - <WX'U  - MU,  - MU>fc  = 4<)VX,U,MU>t 


is  a homogeneous  additive  process.  But 


<WX,U,MU>  = j;X' u (s)  o (x  (s)  ) ds  , 

z 1 0 


which  must  also  have  a representation  as  a homogeneous  additive 
function  of  x(*),  and  which  does  not  otherwise  depend  on  x(0)  = x. 
Thus,  there  is  a Borel  function  i|jU  ( • ) , not  depending  on  x = x(0), 
such  that  f.U,X  (s)  a (x  (s)  ) = i]>U  (x  (s)  ) a (x  (s)  ) , for  almost  all  s 
w.p.l  P^,  each  x.  Q.E.D. 
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